Dynamics of Batch Learning in Multilayer Networks { Overrealizability and Overtraining {

This paper investigates the dynamics of batch learning of multilayer neural networks in the asymptotic case where the number of training data is much larger than the number of parameters. We consider regression problems assuming noisy output data. First, we present experimental results on the behavior in the steepest descent learning of multilayer per-ceptrons and three-layer linear neural networks. We see in these results that strong overtraining, which is the increase of the generalization error in training, occurs if the model has surplus hidden units to realize the target function. Next, to analyze overtraining from the theoretical viewpoint , we consider the solution of the steepest descent learning equation of a three-layer linear neural network, and prove that a network with surplus hidden units shows overtraining. From this theoretical analysis, we can see that overtraining is not a feature observed in the nal stage of learning, but in an intermediate time interval.